Stochastic Contextual Bandits with Long Horizon Rewards

The growing interest in complex decision-making and language modeling problems highlights the importance of sample-efficient learning over very long horizons. This work takes a step this direction by investigating contextual linear bandits where current reward depends on at most s prior actions contexts (not necessarily consecutive), up to time horizon h. In order avoid polynomial dependence h, we propose new algorithms that leverage sparsity discover pattern arm parameters jointly. We consider both data-poor (T= h) regimes derive respective regret upper bounds O(d square-root(sT) +min(q, T) O( square-root(sdT) ), with s, feature dimension d, total T, q is adaptive pattern. Complementing bounds, also show single trajectory brings inherent challenges: While form rank-1 matrix, circulant matrices are not isometric manifolds sample complexity indeed benefits from sparse structure. Our results necessitate analysis address long-range temporal dependencies across data Specifically, utilize connections restricted isometry property formed dependent sub-Gaussian vectors establish guarantees independent interest.